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Executive Summary 



Responding to calls for increased accountability from policymakers, accreditation agencies, and 
the public, colleges and universities are beginning to use evidence of what works to improve 
student success to design, manage, and improve educational programs and services. “Building a 
culture of evidence” to improve student success requires fundamental changes in the way that 
faculty, administrators, and support services staff use student data in decision making. 

Few empirical studies have been carried out on evidence-based decision making in the 
postsecondary sector. The study reported on here, conducted by researchers from the Community 
College Research Center and MDRC, examines what specific data college faculty and 
administrators use in their jobs and the extent to which they use data analysis to design and 
improve the impact of programs and services. The study is based on a survey on the use of 
student data by faculty and administrators at 41 community colleges participating in Achieving 
the Dream: Community Colleges Count. Achieving the Dream is a major national initiative 
designed to improve educational outcomes for community college students, particularly students 
of color, low-income students, and others who have traditionally faced barriers to success in 
college. Twenty-seven of the surveyed colleges joined the Achieving the Dream initiative in 
summer 2004 (Round 1), and 13 joined in summer 2006 (Round 3). In addition to the surveys, 
structured telephone interviews were conducted with administrators and faculty at four of the 
colleges that participated in the survey to explore in greater detail key findings from the survey. 

Achieving the Dream places particular emphasis on two types of data analysis: longitudinal 
analysis and analysis by subgroup. In longitudinal analysis, entering cohorts of students at a 
college are tracked from semester to semester to determine what percentage are achieving critical 
benchmarks, including retention benchmarks, the completion of remedial and college-level 
“gatekeeper” courses, and the completion of college credentials and degrees. Analysis by 
subgroup is used to determine whether students in various ethnic or racial, income, or other 
categories are achieving success at comparable rates or whether there are gaps in their 
achievement levels. Achieving the Dream encourages participating colleges to broadly engage 
faculty, student services staff, and administrators in examining data on student progression, in 
formulating strategies to address achievement gaps, and in evaluating the effectiveness of such 
strategies. Colleges are also expected to use evidence of what works to improve student success 
as the basis for academic program review, strategic planning, and budgeting, and thereby bring 
to scale and sustain proven strategies. 

The survey used in this study, which was administered by the Human Resources Research 
Organization (HumRRO), was conducted over a five-month period beginning in September 
2007. It asked full-time faculty and administrators about what student data they use, how 
accessible data on students are at their college, how they use data in their jobs, and what types of 
data they find most useful. It also asked respondents about their familiarity and involvement with 
Achieving the Dream. The survey received a very favorable response rate, with 60% of faculty 
and 73% of administrators surveyed responding. In fall 2011 we plan to conduct another survey 
of the 13 Round 3 colleges in order to observe changes over time (the survey design allows us to 
match participants’ responses in 2007 with those in 201 1). 
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Results from the present study suggest that relatively high proportions of faeulty and 
administrators at Achieving the Dream colleges use data on student outcomes: 

• More than half of surveyed faculty members reviewed or used data on placement test 
scores, retention rates, graduations rates, and measures of student learning other than 
grades at least once a year. 

• About half of faculty and administrators used data on student achievement gaps on an 
annual basis. 

• About two thirds of faculty and administrators used outside research on effective 
practices at least annually. 

The high rate of data use at the colleges surveyed is perhaps not surprising, given their 
involvement in an initiative premised on the importance of broadly engaging faculty and staff in 
using data and research to improve the quality of programs and services: 

• Faculty and administrators who were involved in Achieving the Dream used data on 
student outcomes more frequently and participated in organized discussions on 
improving student outcomes much more frequently than did their colleagues who were 
not involved in the initiative. 

At the same time, not all participants in Achieving the Dream were heavy data users, and more 
generally, there was considerable variation among individual faculty and administrators in the 
extent to which they used student data. The telephone interviews revealed that there may well 
exist a strong resistance to using data among some faculty. They also suggest that, despite survey 
findings that seem to indicate otherwise, many faculty and administrators may be uncomfortable 
in analyzing data. Contrary to expectation, we did not find much of a correlation between a 
faculty member’s ra nk or length of time having worked at the college and the extent to which he 
or she used data. 

In terms of an overarching commitment to evidence-based decision making, the findings suggest 
that there is only a limited connection between the extent of data use by faculty and 
administrators at Achieving the Dream colleges and the views and management practices of 
college leaders: 

• There is only a weak correlation between various indicators of data use and the extent to 
which respondents indicated that their college overall uses data on outcomes to evaluate 
programs. 

• There is little correlation between the extent to which administrators said that their 
college uses data for program-related decisions and the frequency with which they 
themselves used data. 
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A central premise of Achieving the Dream is that commitment by a college’s leadership and the 
way that a college approaches program evaluation, strategic planning, and budgeting are key to 
encouraging the use of data for improvement by college personnel. Our findings suggest that 
leadership commitment and a data-oriented approach to institutional management may not in 
themselves be sufficient to encourage faculty and administrators to become more data-oriented in 
practice. Additional efforts at the department level are probably needed to change the behavior of 
faculty in particular. Indeed, we found that faeulty in developmental education departments and 
for-eredit occupational programs were more frequent users of data than were faculty in other 
types of departments, particularly those in general edueation. The finding concerning 
developmental faculty is not surprising given that improving developmental instruction has been 
a major focus of Achieving the Dream. It may well be that a similar intensive focus on 
improving outcomes is needed to change practices and to influence the culture in other 
departments. 

In examining the extent of data use among the colleges in our sample, we ranked them by 
institution-level averages on four indicators of data use by faculty and administrators that we 
developed in this study. We did not, however, find much consistency in the rankings by these 
four indicators. This suggests that different colleges in the sample may have emphasized 
different dimensions of data use. In terms of institutional-level results, contrary to our 
expectation, faculty and administrators at larger colleges were not, on average, heavier users of 
data than faculty and administrators at smaller colleges. 

Although eolleges with higher levels of participation in Achieving the Dream by faculty and 
administrators did not exhibit higher average rates of data use, we did find that colleges that 
joined Aehieving the Dream in Round 1 had higher rates of data use than the Round 3 colleges 
on three of the four indicators. This is consistent with (although it does not prove) the hypothesis 
that colleges that have been involved in Achieving the Dream longer are more advanced in their 
use of data in improving student success. It also suggests that engaging faculty and staff in using 
data and building a culture of evidence is a complicated process that requires concerted effort 
over a long period of time. 
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1. Introduction and Background 



Over the past two decades, public education in the U.S. has been under increasing pressure to 
bring about and document improvements in student outcomes. The federal No Child Left Behind 
Act of 2001 requires states to monitor the performance of elementary, middle, and high schools 
based on standardized test scores and to remediate and, if necessary, reorganize or close schools 
that are unable to narrow student achievement gaps. Public higher education institutions have not 
been immune from the push for greater accountability. State and local policymakers, 
accreditation agencies, and boards of trustees are all seeking evidence of improved student 
outcomes. 

An increasing number of public schools and colleges and universities are responding to these 
demands. Borrowing from management techniques first employed in private industry and 
capitalizing on advances in information technology, many are beginning to use data analysis to 
design, manage, and improve educational programs and services. Instituting a “data-driven” or 
“evidence-based” approach to decision making aimed at increasing student success requires 
fundamental change in the way that these educational institutions operate. Proponents sometimes 
refer to this thorough-going organizational change process as “building a culture of evidence.” 

1.1 Research Related to Present Study 

There is a growing body of research on the efforts to use data and research to improve student 
outcomes in public education. Most of the studies to date have focused on the K-12 sector. 
Several explore the conditions under which teachers and administrators in schools use data to 
assess and improve student progress, and provide guidance on how this can be done more 
effectively (Bettesworth, 2006, 2007; Coburn, Toure, & Yamashita, in press; Farley-Ripple, 
2008; U.S. Department of Education, 2004). Other studies have revealed important barriers to 
the effective use of data for improvement in schools, including a lack of quantitative and 
analytical skills on the part of teachers (Wayman & Stringfield, 2006a, 2006b) and absence of 
useful data (Ingram, Louis, & Schroeder, 2004). 

Young (2006) focused attention on how organizational contexts, again in the K-12 sector, can 
positively and negatively influence the use of data in schools. Young concluded that the 
technology environment and the culture of a school, including its norms and values, must change 
in order to successfully use data to improve instruction. Young recommends that districts should 
support principals’ commitment to new practices and recognize that principals require additional 
skills to successfully incorporate such practices into their schools (p. 544). Furthermore, Young 
holds that districts should clarify principals’ accountability for successful implementation of 
data-driven practices. Young also argues that organizational change must influence what teachers 
do on a daily basis so that using data in decision-making becomes accepted as part of the role 
and norms in teaching (p. 545). In another recent study concerning K-12 schools, Coburn and 
Talbert (2006) examined what constitutes valid evidence in decisions about improving student 
success and how considerations of validity are influenced by organizational and individual 
differences. 
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There is growing interest in evidence-based decision making in postsecondary education as well. 
A number of recent publications on the topic, based on informal case studies or on the 
professional experience of the authors, provide encouragement and guidance to colleges and 
universities seeking to build a culture of evidence (Allen & Kazis, 2007; Dwyer, Millett, & 
Payne, 2006; McClenney, McClenney, & Peterson, 2007; Millett, Payne, Dwyer, Stickler, & 
Alexiou, 2008; Petrides, 2004). Some of the recent scholarly work has considered the roles of 
various institutional actors engaged in organizational change. For example, Bensimon (2007) has 
argued that practitioners are in a better position than educational researchers to know how to 
bring about changes in practice that benefit students, particularly those students with a history of 
educational disadvantage and marginalization. According to Bensimon, it is therefore critical that 
practitioners “develop context-dependent knowledge and experience about how to facilitate 
student success” (p. 464). In a review of the literature, Dowd and Tong (2007) examined 
evidence-based decision making and accountability at the postsecondary level; they provide a 
normative argument supporting the idea that data are best used to support student success when 
faculty and staff work together through “evidence-based inquiry councils.” 

There are, however, only a handful of larger-scale empirical studies on evidence-based decision 
making in higher education. In one study, Petrides and Nodine (2005) examined how faculty, 
administrators, and staff in a community college district in California used student outcomes data 
for decision-making. In 2000, the researchers interviewed 70 college personnel and administered 
a survey to 220 faculty, administrators, and staff In 2002, they conducted another study of data 
use in a large, urban community college district. This time they interviewed 27 faculty, 
administrators, and staff about how data retrieval had changed since the implementation of a 
decision support system and a data warehouse in 1998. Their four key findings from these two 
studies are that 1) external accountability mandates do not necessarily result in improved data 
use; 2) even when there are “information bottlenecks” at a college, people find “workarounds” 
that include manual data collection and manipulation and local database creation; 3) even in 
districts that are committed to a data driven environment, significant barriers remain, which 
include concerns about the ability to use technology and data, concerns about “information 
overload,” and concerns that data are perceived as unreliable; and 4) the “information culture” 
and the support for evidence-based decision making within districts is influenced by the district 
leadership. 

In another study, Goldstein (2005) presented results of a national survey of senior IT managers at 
more than 380 postsecondary institutions on the use of data, the technology and tools that 
support data collection and analysis, and the degree to which data support decision making. The 
study found that the use of data in decision making about college programs and practices, or 
what are called “academic analytics,” provides colleges and universities with a competitive 
advantage, at least as perceived by the interview subjects. The research suggests that the factors 
most strongly associated with an institution’s perceived success in implementing academic 
analytics are training, leadership commitment, and staff with strong analytical skills. 

None of the empirical research to date has examined in depth what specific data college faculty 
and administrators use in their jobs and the extent to which they use data analysis to design and 
improve the impact of programs and services. The report presented here offers findings from a 
study designed to fill that gap in the knowledge base. The study was based on a survey and on 
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telephone interviews about the use of student data by faeulty and administrators at eommunity 
eolleges partieipating in Aehieving the Dream: Community Colleges Count. Aehieving the 
Dream is a major national initiative designed to improve edueational outeomes for eommunity 
eollege students, partieularly students of eolor, low-ineome students, and others who have 
traditionally faeed barriers to sueeess in eollege.' 

1,2 Achieving the Dream and Organizational Change 

Aehieving the Dream eneourages eolleges to undertake the following five-step proeess to bring 
about ehanges in poliey and praetiee that lead to improved student sueeess. 

Step 1: Commit to improving student outcomes. The eollege’s senior leadership, with 
support from the board of trustees and faeulty leaders, eommits to making the ehanges in 
poliey and resouree alloeation neeessary to improve student outeomes, eommunieates the 
vision widely within the eollege, and organizes teams to oversee the proeess. 

Step 2: Use data to identify and prioritize problems. The eollege uses longitudinal student 
eohort data and other evidenee to identify gaps in student aehievement. A key premise of 
Aehieving the Dream is that onee faeulty and staff see that eertain groups of students are not 
doing as well as others, they will be motivated to address barriers to student sueeess. To 
ensure that they utilize their resourees to greatest effeet, eolleges are eneouraged to prioritize 
the student aehievement problems that they plan to address. 

Step 3: Engage stakeholders in developing strategies for addressing priority problems. The 
eollege engages faeulty, staff, and other internal and external stakeholders in developing 
strategies for remedying priority problems with student aehievement, based on a diagnosis of 
the eauses and an evaluation of the effeetiveness of previous attempts by the institution and 
others to address similar problems. 

Step 4: Implement, evaluate, and improve strategies. The eollege then implements the 
strategies for addressing priority problems, being sure to evaluate the outeomes and using the 
results to make further improvements. 

Step 5: Institutionalize effective policies and practices. The eollege takes steps to 
institutionalize effeetive polieies and praetiees. Attention is given to how resourees are 
alloeated to bring new initiatives to seale and sustain proven strategies. The proeesses of 
program review, planning, and budgeting are driven by evidenee of what works best for 
students. 

Aehieving the Dream expeets that by following these steps, eolleges will be able to build a 
“eulture of inquiry and evidenee” that will lead to eontinuous improvements in student sueeess. 

Aehieving the Dream’s five-step model shares prineiples with the Total Quality Management 
(TQM) movement and other quality improvement programs sueh as the Baldrige Award and Six 
Sigma. The quality improvement movement in postseeondary edueation is exemplified by the 



’ More information on the initiative is available at www.achievingthedream.org . 
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Academic Quality Improvement Program of the Higher Learning Commission of the North 
Central Association of Colleges and Schools, the Quality Enhancement Plan of the Southern 
Association of Colleges and Schools, and by the Equity Scorecard initative (Bensimon, 2004, 
2005; North Central Association of Colleges and Schools, Higher Eearning Commission, 2008; 
Sallis, 2002; Southern Association of Colleges and Schools, Commission on Colleges, 2008; 

U.S. Department of Commerce, 2006). Each of these models includes a process of data gathering 
to yield a more informed understanding of problem areas, a selection of solutions based on the 
data, and the adoption of an organizational feedback loop in which processes are continuously 
evaluated and improved. 

Over 80 colleges in 15 states are currently involved in the initiative, which is supported by 
Eumina Eoundation for Education and other funders. Participating colleges are asked to undergo 
a year-long process of data analysis and planning, followed by a four-year process in which they 
are expected to implement, evaluate, and, where appropriate, bring to scale strategies for 
improving student success. Achieving the Dream provides financial support to the colleges, 
including a planning grant and an implementation grant totaling $450,000 over five years to 
support data collection and analysis and implementation of improvement strategies. A coach and 
a “data facilitator” are assigned to help each institution, and colleges receive further guidance 
through participation in an annual strategy institute, where they share experiences and findings 
with other colleges. 



1.3 About This Report 

In the study reported on here, researchers surveyed full-time faculty and administrators at 41 
Achieving the Dream colleges in 7 states. These included 28 colleges that joined the initiative in 
the first round (in summer 2004) and 13 colleges that joined in the third round (in summer 2006). 
The survey asked full-time faculty and administrators about what student data they use, how 
accessible data on students are at their college, how they use data in their jobs, and what types of 
data they find most useful. The survey also asked respondents about their familiarity and 
involvement with Achieving the Dream, and it asked for their views more generally on reform 
efforts at their college to improve student success. The survey was conducted over five months 
beginning in September 2007. It received a very favorable response rate, with 60% of faculty and 
73% of administrators surveyed responding. To our knowledge, this is the most extensive survey 
to date on the use of data and attitudes toward data use by college faculty and administrators. In 
fall 2011 we plan to conduct another survey of the 13 Round 3 colleges in order to observe 
changes over time (the survey design allows us to match participants’ responses in 2007 with 
those in 201 1). 

In addition to the surveys, structured telephone interviews were conducted with administrators 
and faculty at four of the colleges that participated in the survey. The purpose of these interviews 
was to explore in more depth key findings from the survey. 

This report is organized as follows. Section 2, following this introduction, presents the 
methodology used in this study. Section 3 describes the main findings on general patterns of data 
use by faculty and administrators at the Achieving the Dream colleges. Section 4 examines 
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factors associated with the use of student outcomes data by faculty and administrators. Section 5 
explores why data use by faculty and administrators is more extensive at some colleges than at 
others. Section 6 presents findings from the in-depth telephone interviews with faculty and 
administrators at four of the colleges. The final section, 7, presents our conclusions and discusses 
the implications of efforts by community colleges to improve student success. 
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2, Methodology 



This section describes the methodology used for the survey of faculty and administrators and for 
the in-depth telephone interviews; these were the main sources of data for the study. 

2,1 Survey of Faculty and Administrators 

To examine patterns of and reasons for data use by faculty and administrators at Achieving the 
Dream colleges, we designed a survey to be delivered using self-administered, on-line 
instruments. 

The CCRC/MDRC research team developed two survey instruments through a process involving 
extensive internal review and revision. The instruments were pilot tested in spring 2007 with 
faculty and administrators at three second-round Achieving the Dream colleges, and were further 
revised based on the responses and feedback from institutional researchers at the pilot colleges. 

The administrator instrument contains 100 items and the faculty instrument contains 101 items, 
most of which overlap. Both instruments ask questions on three main topics: 1) data 
accessibility; 2) data use and usefulness, and 3) familiarity with and involvement in Achieving 
the Dream. Most items use a 7-point Likert scale, although some are yes/no questions. Both 
instruments also ask for demographic information about the respondent and include an open- 
ended question about both the use of data at the respondent’s college and views on Achieving the 
Dream. 

The sampling frame included administrators and full-time faculty members at the colleges that 
joined the Achieving the Dream initiative in Round 1 (in summer 2004) and in Round 3 (in 
summer 2006). Three of the Round 1 colleges declined to participate in the survey, leaving 28 
Round 1 colleges in five states (Florida, New Mexico, North Carolina, Texas, and Virginia) and 
13 Round 3 colleges in two states (Pennsylvania and Washington). 

The survey was administered by the Human Resources Research Organization (HumRRO). 
HumRRO also participated in the final stages of questionnaire development. They were 
responsible for the web programming to permit online administration of the survey, maintenance 
of the online survey, management of the paper versions of the survey and of the telephone 
administration of the survey. They also served as the primary contact for survey questions and 
technical difficulties. In January 2008 they provided CCRC with an interim dataset, and in 
February 2008 they provided the final dataset. 

We asked colleges to provide names, titles, and email addresses for full-time faculty members 
and administrators. For the purposes of this survey, we defined “administrator” as anyone at the 
assistant/associate director level and above, but we allowed the colleges to decide whom to 
include in this category. In cases where a college provided us with all non-faculty positions, we 
edited the list of administrators based on the job titles provided to us. This generated 2,209 full- 
time administrators. 
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We confined the faculty survey to full-time faculty members, since many colleges do not collect 
contact information on adjunct faculty at the college level. Due to the large number of faculty in 
some of the larger colleges involved, we used a single-stage sampling procedure to reduce the 
number of faculty who were invited to participate in the study. We randomly selected 150 faculty 
members at the colleges with faculties larger than 150 by automatically generating random 
numbers for each individual and then selecting the first 150, the minimum number we estimated 
was needed to ensure statistical rigor. We deleted individuals who were no longer affiliated with 
a given college or were on a leave and selected the next person on the list as a replacement. This 
process of replacement continued only during the pre-survey phase, when we were testing email 
addresses; once the survey was deployed, no replacements occurred. Through this process, we 
invited 4,130 full-time faculty members to participate in the study. 

The survey was administered beginning in September 2007. To encourage a high response rate, 
we used a multi-phase administration process. It began with a preliminary email designed to give 
advance notice of the survey and to test the list of email addresses we had received from the 
colleges. The next communication, one week later, was an email with the URL for the survey 
and a unique password for each respondent. The third email was sent as a follow up one week 
later to everyone who had not yet responded. A fourth email was sent one month after the initial 
invitation. One week later, letters with a hard copy of the survey enclosed were mailed to all 
non-respondents. The mailing of the hard copy letters coincided with a request to the college 
presidents to encourage their faculty and administrators to complete the survey. Phone 
administration of the survey to those who had not yet responded began just over two months 
after the initial email was sent. At this time, we were approaching winter break, so we suspended 
efforts to contact the faculty and administrators at the colleges until after the new year. We sent a 
final email to all non-respondents and finished the survey with a final one-week phone 
administration in the week following the February 2008 Achieving the Dream Strategy Institute, 
which teams from every participating college attended. 

The survey achieved a response rate of 60% from the faculty surveyed and a 73% response rate 
from the administrators. Response rates varied considerably by college. Responses to the faculty 
survey ranged from a low of 37% at one college to a high of 85% at another. College response 
rates to the administrator survey ranged from 57% to 93%. Response rates by college are 
provided in Appendix B. 

Respondent demographics of faculty and administrators were fairly similar across colleges. Of 
the faculty respondents, 56% were female, and of the administrators, 62% were female. The 
majority of faculty and administrator survey respondents, more than 78%, identified themselves 
as White. The next largest group comprised those who identified themselves as 
Hispanic/Latino/Spanish; they made up 12% of the faculty and 15% of the administrators. Black, 
Asian or Pacific Islander, and American Indian or Alaskan Native faculty comprised 8%, 3 %, 
and 2% of the faculty respondents, respectively. Black administrators made up 12% of the 
administrator respondents. Asian or Pacific Islanders and American Indian or Alaskan Natives 
made up 3% and 1% of administrator respondents, respectively. Ten percent of faculty and 8% 
of administrators identified themselves as “other.” 
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More than 62% of the faculty who completed the survey taught in academic transfer/general 
education areas. Administrators involved in academic affairs and instruction made up 20% of the 
respondents, and those in student affairs/student services accounted for 18% of the respondents. 
The institutional research/effectiveness staff accounted for just over 3% of the respondents. 
Additional demographic information is available in Appendix C. 

2,2 In-Depth Telephone Interviews 

To explore the survey findings on the patterns and determinants of data use in greater detail, we 
conducted in-depth telephone interviews with key personnel at four colleges that participated in 
the survey. 

These four colleges were selected on the basis of four criteria. First, the colleges are all first- 
round participants in the initiative. This means that they have been exposed to the concept of a 
culture of evidence for almost four years. This was important because creating a culture of 
evidence requires fundamental organizational change, so colleges needed to have had time to 
adopt such a culture. 

The second and third criteria influenced the selection of the states in which the colleges are 
located. Note that the Round 1 Achieving the Dream colleges were drawn from five states: 
Florida, New Mexico, North Carolina, Texas, and Virginia. The states chosen for telephone 
interviews were selected on the basis of their accrediting region. New Mexico and Virginia were 
selected because they are in regions with accrediting agencies that were the earliest to adopt 
guidelines relating to the use of data and evidence to improve student outcomes. We 
hypothesized that the longer the accrediting region had these guidelines in place, the stronger the 
pressure on colleges in these states would be to use data to support improving student outcomes. 
New Mexico is under the North Central Association of Colleges and Schools’ Higher Learning 
Commission (NCA-HLC). Virginia is a member of the Southern Association of Colleges and 
Schools (SACS). NCA-HLC and SACS were the first of the regional accreditation agencies to 
have focused attention on the use of evidence for improving student outcomes (Biswas, 2006). 
Also, both states have adopted performance funding policies within the past three years. Such 
policies require colleges to report performance data, and they tie a certain percentage of funding 
for colleges to their performance. Together, these criteria were important because they suggest 
that these are states in which colleges are strongly encouraged to use data to document and 
improve performance. 

Finally, the colleges selected represent a range of sizes within each state. We hypothesized that 
smaller colleges would have an easier time promoting data use as part of the initiative because 
there would be fewer communication and coordination issues. 

Once the colleges were selected we identified potential interviewees at each college based on the 
functions they carried out. They did not need to be active participants in Achieving the Dream. 
Persons we interviewed included the administrative leadership of the college, including the 
president and vice presidents, the Achieving the Dream coordinator, institutional research staff, 
information technology staff, student services personnel, and a mix of faculty teaching 
developmental and non-developmental education courses. We created interview protocols for 
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each position and asked questions about interviewees’ use of data, their perceptions about the 
decision-making process at the college, and the role of Achieving the Dream and other initiatives 
in supporting data-based decision making at each college. These interviews were conducted in 
the summer and fall of 2008. 

We interviewed 17 faculty members and 25 administrators. The interviewees included a mix of 
individuals who had and had not been directly involved with the Achieving the Dream initiative. 
The faculty we interviewed taught either developmental or college-level math or English. The 
senior administrators we interviewed included, at each college, the senior academic officer, the 
senior student services officer, the director of institutional research, the Achieving the Dream 
coordinator, and at least one member of the student services staff. 
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3. Patterns of Data Use by Faculty and Administrators 



This section presents descriptive statistics of survey responses from faculty and administrators at 
Achieving the Dream colleges to questions concerning the use of data and research on students 
by themselves, their departments, and their colleges. 

3.1 Use of Data by Faculty and Administrators 

Frequency of data use by type 

Use of student outcomes data by faculty and administrators at the surveyed Achieving the Dream 
colleges was perhaps more widespread than expected based on our earlier findings in a baseline 
evaluation of first-round Achieving the Dream colleges (Brock et ah, 2007) and based on 
feedback from the coaches and data facilitators about the colleges’ use of data and research. 

Table 1 shows the percentages of faculty and administrators who indicated using or reviewing 
various sorts of information at least once a year as well as the percentage who said they never 
use a given type of information. Not surprisingly, high percentages of faculty reviewed or used 
grades and course evaluations at least once a year, while the majority of administrators reviewed 
information on college finances at least annually. What might be more unexpected is that more 
than half of faculty used data on placement test scores, retention rates, graduation rates, and 
measures of student learning other than grades, among other types of student outcomes data, 
once a year or more. Still, over a third of faculty never used measures of student learning other 
than grades and never looked at information on students broken down by race or ethnicity, and 
nearly a third never reviewed data on student achievement gaps. 

Most faculty and administrators reviewed enrollment data at least once a year. About two thirds 
of faculty and administrators used outside research on effective practices at least annually. About 
half of faculty and administrators reviewed or used data on student achievement gaps on at least 
an annual basis. Administrators were more likely than faculty to use research produced by their 
own college. 
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Table 1. How often do you review or use the following types of information in your job? 





Percentage of Respondent Group Using the Given 
Information Type* 




At least once per year 




Never 


Type of Information 


Faculty 


Admins. 


Faculty 


Admins. 


Placement test scores** 


69.3 


49.4 


23.8 


44.9 


Enrollment data 


87.3 


83.6 


CO 

CO 


14.0 


Grades 


89.9 


61.3 


7.1 


34.3 


Course evaluations 


89.4 


46.4 


5.3 


47.6 


Measures of student learning other 
than grades** 


54.7 


45.5 


34.3 


47.0 


Retention rates** 


72.0 


65.3 


17.5 


29.0 


Graduation rates** 


64.5 


64.0 


22.7 


29.5 


Transfer rates** 


46.4 


50.5 


35.6 


40.7 


Percentage of students successfully 
completing developmental ed.** 


47.8 


48.4 


37.9 


42.9 


Financial aid** 


35.2 


52.6 


52.1 


39.1 


College budget and finances 


51.2 


75.3 


34.9 


18.9 


Results from external surveys** 


41.8 


51.7 


39.2 


36.9 


Focus groups or other qualitative 
data** 


40.1 


51.5 


37.1 


33.2 


Research by the college** 


58.1 


68.2 


21.5 


20.2 


Outside research on effective 
practices 


69.0 


65.9 


16.4 


23.4 


Data on student achievement 
gaps** 


50.7 


49.7 


30.8 


39.1 


Information broken down by 
students’ race or ethnicity** 


47.0 


58.7 


35.6 


31.7 


Information broken down by 
students’ income levels or receipt 
of financial aid** 


32.2 


48.5 


51.7 


41.1 



* “Type of Information Not Available” responses were treated as missing. 
** Types of data that are promoted by Aehieving the Dream. 
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Perceived usefulness of data by type 



The majority of faculty and administrators surveyed found most of the types of information listed 
below in Table 2 at least somewhat useful in their jobs. A slightly higher share of administrators 
than faculty found these types of information at least somewhat useful in their jobs, but the 
percentages are generally high for both faculty and administrators. Two thirds of the faculty 
respondents indicated that research reports and other information their college provides are 
generally helpful to their work as teachers. 



Table 2. In your role as a faculty member (administrator), rate tbe usefulness of tbe 



following types of information to your job. 





Percentage of Respondent Group Indicating Type of Info 
is “Somewhat Useful” to “Very Useful”* 


Type of Information 


Faculty 


Administrators 


Placement test scores 


74.1 


83.4 


Enrollment data 


75.1 


94.8 


Grades 


84.7 


85.4 


Course evaluations 


89.9 


82.9 


Measures of student learning other than 
grades 


70.9 


82.0 


Retention rates 


80.6 


89.5 


Graduation rates 


70.8 


87.7 


Transfer rates 


65.3 


83.1 


Percentage of students successfully 
completing developmental ed. 


63.7 


82.9 


Financial aid 


41.9 


81.8 


College budget and finances 


50.3 


92.9 


Results from external surveys 


54.0 


81.3 


Focus groups or other qualitative data 


60.4 


83.8 


Research by the college 


66.7 


88.6 


Outside research on effective practices 


79.1 


87.6 


Data on student achievement gaps 


69.3 


82.1 


Information broken down by students’ 
race or ethnicity 


50.9 


80.6 


Information broken down by students’ 
income levels or receipt of financial 
aid 


45.9 


80.0 



* “Not Applicable” responses treated as missing. 
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Use of data and research by faculty in teaching-related decisions 



As is evident from Table 3 below, the majority of faculty at Achieving the Dream colleges use 
data and research at least to some extent in decisions related to teaching. Around one in five is a 
heavy user of data and research for teaching decisions. Only around one in ten use data and 
research “not at all.” 

Table 3. How much do you use data and research on students in your own decisions 



about the following? 





Percentage of Faculty Who Use Data and Research for the 
Given Decision Type 


Decision Type 


At least some 


A lot 


Not at all 


Curriculum 


76.2 


18.3 


14.1 


Teaching practices 


82.2 


24.2 


9.0 


Advising students 


80.7 


23.3 


11.3 


Identifying students who are 
stmggling academically 


78.2 


22.3 


11.2 



Participation in organized discussions on improving student success 

The majority of faculty and administrators at Achieving the Dream colleges indicated that they 
participate at least once a year in organized discussion on improving student success (see Table 
4). Fewer participate in discussions about the needs or performance of students of color or of 
low-income students in particular than about improving academic achievement of students more 
generally. Approximately one third of faculty respondents indicated that they never participate in 
discussions about the performance of students of color or low-income students. 

In a question not reflected in Table 4, two thirds of faculty respondents agreed that asking faculty 
members to regularly participate in discussions about data on student outcomes is a good use of 
their time. Fewer than 20 percent (18%) disagreed with this. 

Table 4, How often have you participated in organized discussions at the college on topics 



related to improving student success? 





Percentage of Respondent Group Participating in 
Discussions on the Given Topic 




At least once per year 


Never 




Topic of Discussion 


Faculty 


Admins. 


Faculty 


Admins. 


Improving academic achievement 
or closing achievement gaps 


78.0 


63.9 


9.5 


26.8 


Academic needs or performance of 
students of color 


56.2 


57.0 


26.5 


34.4 


Academic needs or performance of 
low-income students 


55.9 


59.2 


26.5 


32.4 
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Use of data by academic departments 

Most faculty who responded to the survey indicated that they were in departments that use data 
and research for programmatic decisions at least to some extent (see Table 5 below). Around one 
in four were in departments that are heavy users of data to make program decisions. Only a small 
percentage of the faculty respondents were in departments that do not make use of data and 
research for such decisions. 

In a question not reflected in Table 5, four out of five faculty respondents indicated that their 
department meets at least one a year to discuss the success rates of students in the courses they 
teach. A similar majority (80.2%) are in departments that regularly evaluate the effectiveness of 
new academic programs, projects, or practices. 

Table 5. How much does your department use data and research on students in 



decision-making about the issues below? (Question asked of faculty only.) 



Percentage of Faculty Whose Department Uses Data and 
Research for Decisions on the Given Topic* 


Issues 


At least some 


A lot 


Not at all 


Curriculum 


79.2 


25.2 


11.1 


Teaching practices 


81.5 


22.6 


9.6 


Tutoring or other academic support 


79.4 


22.3 


10.4 


Program planning 


83.0 


25.5 


7.7 


Academic program review or 
evaluation 


84.5 


27.0 


6.9 


Long-term strategic planning 


80.4 


22.4 


8.8 


Budgeting or resource allocation 


71.1 


17.6 


14.6 


Identifying and redesigning high- 
failure-rate courses 


51.8 


21.5 


15.2 



* “Don’t Know” responses treated as missing. 



Use of data by college 

The survey asked administrators to assess how much their college uses data and research on 
students in decision-making. As in evident from Table 6 below, the majority of respondents 
indicated that their college uses data and research on students in decision-making on program 
and planning issues at least to some extent. A third or more indicated that their college uses data 
and research extensively. Only a small fraction indicated that their college does not use data and 
research in decision-making. 

In a question not reflected in Table 6, the vast majority of administrators (91.0%) also indicated 
that their college uses data on student outcomes (e.g., persistence, learning, or degree 
attainment), not just enrollments, to evaluate academic programs and departments. A similar 
pereentage (92.5%) indicated that each department or division in their college is required to set 
measurable goals and objectives as part of the planning process. Three fourths (75.3%) of 
administrators said that budget requests at their college must be supported by evidence that 
students will benefit as a result. 
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Table 6. How much does your college use data and research on students in decision-making 



about the issues below? (Question asked of administrators only.) 



Percentage of Administrators Indicating Their College 
Uses Data and Research for Decisions on the Given Topic* 


Issue 


At least some 


A lot 


Not at all 


Curriculum 


88.1 


31.6 


4.5 


Program planning 


60.1 


32.9 


3.6 


Academic program review or 
evaluation 


56.6 


36.5 


3.2 


Long-term strategic planning 


55.0 


38.6 


3.2 


Budgeting and resource allocation 


55.5 


36.2 


4.2 


Identifying areas for improvement 
at the college 


55.4 


38.3 


2.7 


* “Don’t Know” responses treated as 


missing. 






Influence on the use of data 








The survey asked administrators about the sources of influence on the use of data by themselves 
and their colleagues at the college. As shown in Table 7 below, the majority of administrators 
responding to the survey believed that accreditation, Achieving the Dream, and the college’s 
senior leadership have had a lot of influence on data use at their institution. A near majority 
believed that state mandates and other grant programs have had a lot of influence on data use at 
their college. Only a quarter of administrators indicated that their college’s trustees have had a 
lot of influence on the use of data. 


Table 7. How much influence have tbe following bad in promoting data use by college 
administrators at your college? (Question asked of administrators only,) 


Percentage of Administrators Indicating the Given 
Influence Has Had on Data Use at College* 


Possible Influences 


At least moderate 
influence 


A lot of influence 


No influence 


College leadership (chancellor, 
presidents, vice presidents) 


92.9 


56.0 


2.7 


College board of trustees 


73.8 


25.4 


11.9 


State mandates 


93.5 


46.6 


2.5 


Accreditation reviews 


97.2 


63.6 


1.4 


Achieving the Dream 


95.9 


59.0 


2.1 


Other grant programs (e.g.. Title 
111) 


95.2 


45.4 


1.9 


* “Don’t Know” responses treated as 


missing. 
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3.2 Accessibility of Data and Perceived Barriers to Data Use 



Sources of data on students 

Faculty and administrators at the Achieving the Dream colleges indicated that they use a variety 
of sources or methods to get information on groups of students (see Table 8 below). About a 
third of faculty members do searches themselves using their college’s student information system 
or their college’s website or fact book. Similar proportions make requests for information to the 
college’s IR or IT offices or use reports distributed by the college. Administrators are even more 
likely than faculty to search for information themselves, although the largest percentages make 
requests to IR or use reports from the college. Fewer than one in ten faculty members and one in 
four administrators indicated that they generally do not need information on groups of students. 



Table 8. When I need information about groups of students (i.e., more than one 

student at a time), I use tbe following sources. (Respondents were asked to 
circle all that apply.) 





Percentage of Respondent Group Indicating That 
They Use the Given Source 


Source 


Faculty 


Administrators 


Searches using the college’s student 
information system 


32.9 


41.1 


Data from the college’s website or fact 
book 


30.9 


43.2 


Reports distributed by the college’s 
institutional research (IR) office or 
other departments 


32.7 


50.0 


Requests to the IR or information 
technology (IT) staff 


32.1 


53.6 


My department’s database 


24.2 


29.5 


State databases or research reports 


22.3 


25.4 


I generally do not need information 
about groups of students 


9.7 


23.1 



A majority of faculty and administrators seem satisfied that they are able to access information 
they need in a timely manner and that the information they receive is accurate (see Table 9 
below). About half of faculty respondents (51.8%) indicated that the research reports and other 
information the college provides to faculty are generally helpful to their work as teachers. Only 
about a quarter (25.6%, not shown in the table) disagreed with the statement that such 
information is helpful to them in their teaching roles (the other quarter was neutral, neither 
agreeing nor disagreeing). 
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Table 9. To what extent do you agree or disagree with the following statements? 



Percentage of Respondent Group Indicating 
They “Agree” to “Strongly Agree” 




Faculty 


Administrators 


The data in the college’s student 
information system are generally accurate 
and error free. 


63.9 


70.7 


The data I need are generally available in a 
user-friendly format. 


57.9 


62.1 


The college’s institutional research staff is 
responsive to requests for information. 


70.1 


77.5 


The college’s institutional research staff is 
adequately staffed for the college’s 
information and research needs. 


49.2 


52.9 


The reports and other information the 
college provides to administrators and 
faculty are typically clear and easy to 
follow. 


62.8 


72.3 


I am able to obtain the information I need 
in a timely fashion. 


62.4 


69.1 


The research reports and other information 
the college provides to faculty are 
generally helpful to our work as teachers. 


51.8 


— 



* “Don’t Know” and “Not Applicable” responses treated as missing. 



Reasons for not using data 

As shown in Table 10, about a third of faculty indicated that one reason they do not use data and 
research is that they are too busy with their teaching responsibilities. Other than that, most 
faculty seem to have thought that using data and research on students is part of their 
responsibility as a faculty member and that they have the skills needed to analyze data. 

Over 30% of administrators indicated that it is not their responsibility to use data and research on 
students. This may be because our sample of administrators included some in athletics and other 
areas where information on students is generally not used. The fact that a similar percentage of 
administrators indicated that the data available are not relevant to their roles supports this 
conclusion. 
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Table 10. I generally do not use data and research on students for the following 



reasons. 





Percentage of Respondent Group Indicating 
They “Agree” to “Strongly Agree” 


Reason 


Faculty 


Administrators 


I am too busy with my teaching 
responsibilities. 


32.0 


8.7 


It is not part of my responsibilities as a 
faculty member/administrator. 


13.4 


31.0 


I do not have the research skills to 
understand and use data and research. 


16.5 


10.6 


I do not trust the data that are available. 


16.2 


11.2 


The data that are available are not relevant 
to my role as a faculty member/ 
administrator. 


21.4 


31.9 



Relevant training 

More than a third of faculty (35.2%) said that they have been involved in training or professional 
development in the past year on institutional research or data analysis, and over half (57.9%) said 
they have participated in training or development on program evaluation or assessment (see 
Table 11). This and the corresponding statistics for administrators were higher than we expected. 



Table 11, Have you been involved in any training or other professional 
development in the past year related to the following topics? 





Percentage of Respondent Group Indicating 
“Yes” 


Topic 


Faculty 


Administrators 


Institutional research and/or data analysis 


35.2 


37.2 


Program evaluation and/or assessment 


57.9 


40.6 
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4, Correlates of Data Use by Faculty and Administrators 



As is evident from the last section, the survey results reveal considerable variation in the extent 
to which individual faculty members and administrators at Achieving the Dream colleges use 
data in their jobs. This section examines factors associated with the tendency of faculty and 
administrators to use (or not use) student data in their work. The factors examined include, 
among others, attitudes about data use, perceptions of data accessibility and usefulness, 
department and college policies, and participation in Achieving the Dream. Similar analyses 
were conducted on factors associated with the extent to which faculty and administrators 
participate in organized discussions on ways to improve student achievement. 

4,1 Indicators of Data Use 

Both the faculty and administrator surveys contained multiple questions about the use of data and 
participation in organized discussions related to improving student success. To simplify the 
analysis of factors associated with the behaviors of interest, we created four composite indicators 
of data use by faculty and administrators. 

1) Use of data on student outcomes. 

2) Use of data disaggregated by student race, ethnicity, or income. 

3) Participation in organized discussions on improving student success. 

4) Use (by faculty) of data and research in teaching-related decisions. 

Appendix A lists the specific survey items used to create each indicator and outlines the method 
we employed to do so. 

Tables 12a and 12b below show the correlations among these four indicators for faculty and 
administrators respectively. Generally, the correlations among the indicators are fairly strong. 
Faculty and administrators who use student outcomes data frequently also use disaggregated 
student data and participate in organized discussions about improving student achievement more 
frequently. The correlation is somewhat weaker, though still positive, between use by faculty of 
data in teaching decisions and the indicators of use of outcomes data and participation in 
discussions about improving achievement. 
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Table 1 


12a. Correlation matrix of composite indicators of data us 




1 


2 


3 


4 


1 


1.00 


2 


0.64 


1.00 






3 


0.44 


0.46 


1.00 




4 


0.44 


0.30 


0.32 


1.00 



Table 


2b. Correlation matrix of composite indie 




1 


2 


3 


1 


1.00 


2 


0.69 


1.00 




3 


0.64 


0.60 


1.00 



Key for Tables 12a and 12b: 

1 . Frequeney of use of various sorts of data related to student outeomes. 

2. Frequeney of use of disaggregated student data. 

3. Frequeney of partieipation in organized diseussions about improving student aehievement. 

4. Extent of use by faeulty of data to inform teaehing-related deeisions (faeulty survey only). 



To get a sense of why some faculty and administrators might use data on student outcomes and 
engage in discussions about improving student achievement more than others, we calculated the 
correlations between the four composite indicators of data use described above and various 
factors that we hypothesized are associated with use of data by the faculty and administrators in 
our sample. These factors can be grouped in the following categories. 



• Involvement in Achieving the Dream 

• Perceived usefulness of student data 

• Perceived accessibility and quality of student data 

• Reasons for not using data 

• Attitudes about the potential of students to succeed and the college’s responsibility in 
helping students succeed 

• Participation in training on data analysis, assessment, or program evaluation 

• Academic department practices on use of data and program evaluation and improvement 

• College/administrative department practices and leadership for data-driven decision 
making and student success 

• Influence of external factors in promoting data use by college administrators (administrator 
survey only) 

• Academic program area (for faculty) or administrative functional area (for administrators) 

• Length of time at the college, age, and (for faculty) rank 

• Respondent’s demographics. 

See Appendix A for a list of the specific measures examined under each category. 
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4,2 Findings on Correlates of Data Use 

Table 13 presents a summary of findings on the correlation between various factors we 
hypothesized to be related to data use and the four indicators of data use. Detailed statistics on 
correlations (for continuous factor measures) and differences of mean responses (for categorical 
measures) are presented in an Excel document that is available from the authors. 

We want to stress that in this analysis we are merely able to examine the correlation between 
particular factors and the indicators of data use. This analysis cannot tell us if the relationship is 
causal. So, for example, although we find that participation in training on data analysis and 
program assessment is positively correlated with the indicators of data use, this does not 
necessarily mean that colleges can increase data use by increasing the amount of training 
provided. It could be that faculty and administrators who are heavier users of data are more 
likely to seek out training in data use. 

In the text that follows the large table below, we examine in more detail the correlations between 
the various factors and the indicators of data use. 
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Table 13. Factors correlated with indicators of data use by faculty and administrators: Summary of findings 
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Involvement in Achieving the Dream 

Faculty and administrators who were involved in Achieving the Dream, either in the core or data 
teams or in implementing strategies, used data on student outcomes more frequently and 
participated in organized discussions on improving student outcomes much more frequently than 
did their colleagues who were not involved in the initiative. This is not surprising given that 
colleges participating in Achieving the Dream are encouraged to broadly engage faculty, staff, 
and administrators in examining data on student progression and outcomes, and to work together 
to design, implement, and evaluate efforts to improve student achievement, particularly among 
groups of students who have faced barriers to success in the past. 

Faculty members participating in Achieving the Dream were much more likely than non- 
participating faculty to make more frequent use of data that are of particular interest to 
Achieving the Dream. Thus, for example, the differences between participating and non- 
participating faculty in the frequency with which they reviewed data on developmental education 
completion rates and academic achievement gaps are greater than the differences for other 
student outcomes data for measures that are not central to Achieving the Dream. A similar 
pattern is evident in the rates of student outcomes data use between participating and non- 
participating administrators. Participation in Achieving the Dream is also associated with a 
greater tendency of faculty and administrators to use data disaggregated by race or ethnicity and 
by income or receipt of financial aid. 

We hypothesized that faculty members and administrators who believed that Achieving the 
Dream would have a lasting impact on college practice would be more likely to use data than 
would faculty who believed that Achieving the Dream is just another “flavor of the month” 
whose effects will fade quickly when the grant ends. However, we found only weak correlations 
(< 0.23) between measures of faculty and administrator opinions about the sustainability of 
Achieving the Dream and the indicators of data use. 

As mentioned above, these results simply show correlation, not causality. Thus, although we 
found that participation in Achieving the Dream is correlated with greater frequency of data use, 
we cannot say definitively that involving more faculty and administrators in Achieving the 
Dream will result in higher rates of data use. 

Perceived usefulness of student data 

Not surprisingly, faculty and administrators who had positive views about the usefulness of 
various sorts of student outcomes data (such as learning measures, retention and graduation rates, 
and achievement gaps) used student outcomes data more frequently than did those who question 
the usefulness of such data. Faculty who viewed data positively were specifically more likely to 
use data and research to inform decisions related to their teaching. For faculty, the correlation 
between the perceived usefulness of student outcomes data and the measure of the use of data in 
teaching-related decisions is especially strong. 
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Perceptions of the usefulness of student outcomes data are also correlated with the frequency of 
participation by faculty and administrators in organized discussions with peers about improving 
student achievement and the frequency of using data disaggregated by race, ethnicity, or income, 
but the correlations are weaker than those associated with indicators of the frequency of data use. 

Faculty who believed that institutional research is useful to their work as teachers were 
somewhat more likely to use data and participate in discussions about improving student 
outcomes than were faculty who thought otherwise, but the correlation is not strong (ranging 
from 0.21 to 0.34 across the indicators of data use). 

Perceived accessibility and quality of data on students 

Surprisingly, there is little correlation (< 0.2) between the perceptions of faculty and 
administrators about whether or not data on students were accurate and accessible and the 
frequency with which they used data on student outcomes or participated in organized 
discussions on improving academic achievement. This suggests that faculty and administrators 
who use data are motivated to do so regardless of whether the data are readily accessible or not. 

It is also possible that faculty and administrators get their data from the institutional research 
office and therefore do not have to deal with the issue of data accessibility. However, this 
explanation alone does not account for the weak correlation between perceptions of accuracy and 
frequency of data use. 

Reasons for not using data 

The survey asked faculty and administrators about various possible reasons they might not use 
data on students, such as being too busy, not feeling responsible for analyzing student data, or 
not having the skills to do so. For both faculty and administrators, those who felt more strongly 
about reasons they might not use data were, as expected, less likely to use data as measured by 
all four indicators. 

The reasons most associated with a lower tendency to use data were that using student data was 
not a job responsibility or that the data were irrelevant to their jobs. This was even more the case 
for administrators than for faculty. Perhaps this is the result of the fact that we included in our 
survey all senior level administrators, including those in areas such as athletics and finance, who 
may have been less concerned with student data as part of their jobs than administrators in 
academic or student affairs. 

Faculty and administrators who indicated that they do not have the skills to analyze data or that 
they do not trust their college’s data were also less frequent users of data, although the 
correlations are weak. Similarly, faculty who indicated that participating in discussions about 
improving student success was not a good use of their time were less frequent users of data on all 
four indicators, although here again the correlation is weak. 
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Attitudes about the potential of students to succeed and the college ’s responsibility in helping 
students succeed 

We hypothesized that faculty and administrators would be more likely to use data on students if 
they believed that their college could actually do much to improve student outcomes and should 
be doing so. However, we found only a weak correlation between the four indicators of data use 
and faculty and administrator beliefs about the potential of all students to succeed academically. 
We found no correlation between the indicators of data use and the extent to which respondents 
believed that it is appropriate for their college to provide extra help to certain groups of students 
or felt that they and their colleagues could do more to help students succeed. 

Length of time at the college, age, and (for faculty) rank 

We hypothesized that junior faculty would be more likely than senior faculty to use data on 
student outcomes because junior faculty might be more comfortable with technology and 
because they are still learning what is most effective for them in the classroom. However, the 
only statistically significant difference we found across the four indicators of data use is that 
assistant professors used data on student outcomes less frequently than did full professors. 

Also contrary to expectation, there does not seem to be a correlation between the length of time 
faculty members or administrators have been at their college or their age and their use of student 
data. 

Surprisingly, administrators who were over 55 were much more frequent users of student 
outcomes data than those under 35. Administrators over 55 were also more frequent users of 
student data disaggregated by race, ethnicity, or income. Administrators over 35 were more 
likely than those under 35 to participate frequently in organized discussions about improving 
student achievement. Interestingly, administrators 65 or older were most likely to participate in 
discussions about improving student achievement. 

Gender and race/ethnicity 

Female faculty members indicated that they used data and research in teaching-related decisions 
more frequently than did male faculty members and participated in organized discussions about 
students more frequently than did their male counterparts. However, there is no statistically 
significant difference in the frequency with which male and female faculty used data on student 
outcomes. 

White faculty were less likely than non- White faculty to frequently use data on student 
outcomes, to participate in organized discussions on student achievement, to use data and 
research in teaching-related decisions, and to use data disaggregated by race, ethnicity, and 
income. Faculty who classified themselves as Black were more frequent participants in 
organized discussions on improving student achievement. Black and Native American faculty 
used data broken out by race, ethnicity, and income more frequently than did faculty of other 
races and ethnicities. Interestingly, Asian/Pacific Islander faculty used data in teaching-related 
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decisions more frequently than did faculty of other racial and ethnic groups. It is also notable that 
there does not seem to be any correlation between Hispanic faculty and use of data. 

In contrast to the patterns observed for faculty, White administrators were more frequent users of 
student outcomes data than were non- White administrators. Hispanic administrators were less 
likely to participate in organized discussions on improving student achievement than were non- 
Hispanics. 

Participation in training on data analysis, assessment, or program evaluation 

Faculty and administrators who had received training or professional development on analyzing 
data or on program evaluation were, not surprisingly, more likely than faculty members who had 
not received such training to use data across all the indicators. Again, our analysis only shows 
correlation, not causality, so it is not necessarily the case that colleges will increase use of data if 
they train more faculty and administrators on data use and program assessment. That could 
happen, but it could also be the case that faculty and administrators who are already motivated to 
use data pursue training that they think will aid them in doing so. 

Academic department practices on use of data and program evaluation and improvement 

Faculty in academic departments that use data and research in departmental decision making 
themselves, not surprisingly, used data and research more frequently in teaching-related 
decisions than did faculty in departments where data-based decision making is not so prevalent. 
Faculty in departments that use data for decisions more extensively also used data on student 
outcomes and participated in organized discussions on improving student success, although with 
these indicators, the correlation with department practices is not as strong. 

Similarly, faculty in departments that meet at least once a year to discuss student success rates 
and that evaluate the effectiveness of programs were more likely to use data as measured by the 
four indicators employed here. The mean responses indicate that faculty who reported that their 
departments meet at least once a year to discuss the success rates of their students reviewed 
and/or used student outcomes data at least once every two to three years, while those in 
departments that meet less frequently used student outcomes data less frequently. Meeting with 
colleagues as a department at least once a year, not surprisingly, is correlated with more frequent 
participation in organized discussions about academic achievement and a greater tendency 
among faculty to use data and research on students in their own decisions about teaching. 

Academic program area (for faculty) 

Faculty in general education were on average less likely than faculty in other program areas to 
use data on student outcomes and to use data and research in decisions related to their teaching 
on a frequent basis. In contrast, faculty who teach in developmental or for-credit occupational 
programs were more likely than those in other fields to do so. Developmental faculty members 
were also significantly more likely to participate in organized discussions on student 
achievement and to use data disaggregated by race, ethnicity, or income. Adult basic education 
faculty used data no more frequently than faculty in other areas. Interestingly, even though they 
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were more likely than faculty in other areas to use data in teaching-related decisions, faculty in 
for-credit occupational programs were less likely to participate in organized discussions about 
student achievement or to use data broken down by race, ethnicity, or income. Faculty in non- 
credit occupational programs used data related to student outcomes more frequently than did 
faculty in other fields, but there are no statistically significant differences in the rates of data use 
between these faculty and others on the other three indicators of data use.. 

Not surprisingly, we found that faculty members who also have administrative roles were more 
likely than faculty not involved in administration to use data across the four measures. 

Administrative functional area (for administrators) 

Table 14 below ranks administrative functional areas by the three indicators of data use 
applicable to administrators. It is not surprising that institutional research administrators were the 
most frequent data users on all three indicators. Academic affairs administrators were more 
frequent users of data than were those in student affairs/services. (It is not clear how “academic 
affairs” differs from “instruction.”) Interestingly, administrators in external affairs were more 
frequent users of data on student outcomes than were student affairs administrators. External 
affairs administrators were also more likely to disaggregate data by race, ethnicity, or income 
than were administrators in both student and academic affairs. 



Table 14. Rank of administrative functions by three measures of data use 



Rank 


Frequency of use of 
student outcomes data 


Frequency of 
participation in 
organized discussions 
on improving student 
success 


Frequency of use of 
data disaggregated by 
race/ethnicity or 
income 


1 


Institutional research 


Institutional research 


Institutional research 


2 


Academic affairs 


President’s office 


External relations 


3 


President’s office 


Academic affairs 


President’s office 


4 


External relations 


Student affairs/services 


Academic affairs 


5 


Career and technical ed. 


Instruction 


Student affairs/services 


6 


Student affairs/services 


Career and technical ed. 


Admissions 


7 


Instruction 


External relations 


Career and technical ed. 


8 


Admissions 


Admissions 


Instruction 


9 


Continuing education 


Registration 


Einance 


10 


Information technology 


Athletics 


Continuing education 



Interestingly, administrators who also had a faculty role were more likely to use data on all three 
measures. 

College/administrative department practices and leadership 

There is only a weak correlation between the various indicators of data use by individual faculty 
and the extent to which respondents indicated that their college overall uses data on student 
outcomes to evaluate programs. Even weaker is the correlation between faculty data use and 
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faculty members’ perceptions about the level of commitment by the college’s leadership to 
making decisions based on data and the clarity of the leadership’s vision on how to inerease 
student academic success. These findings and the earlier ones about departmental practices 
suggest that the practiees of individual academic departments have a greater bearing on the use 
of data by faculty members than do those of the college overall. 

There is also surprisingly little correlation between the extent to which administrators said that 
their college uses data for program-related decisions and the frequency with which they 
themselves used data. 

Administrators at institutions that evaluate the effectiveness of educational programs and 
services or that use data on student outcomes to evaluate academic programs and departments 
used data and engaged in organized discussions about improving student achievement more 
frequently than did administrators at colleges that do not evaluate programs and services, but the 
differences are not statistically significant. Similarly, there does not seem to be much correlation 
between the frequency with which administrators used student outcomes data or engaged in 
organized discussions on improving student outcomes and whether or not: 

• The college has a strategic plan; 

• Departments and divisions are required to set measurable goals as part of the planning 
process; 

• The college requires that budget requests be supported by evidence that students will 
benefit. 

Curiously, the three practices above are associated with a lower frequency of use among 
administrators of student data disaggregated by race, ethnicity, or income. It is not clear why 
colleges with established systems for strategic planning and budgeting would be less likely to use 
data disaggregated by race or income than would colleges without such systems. 

Influence of external factors in promoting data use by college administrators (administrator 
survey only) 

We asked administrators about the extent to which various internal and external influences have 
promoted the use of data by the college. These included college leadership, board of trustees, 
state mandates, accreditation reviews. Achieving the Dream, and other grant programs. 
Surprisingly, there is only a very weak correlation (< 0.1) between administrators’ ratings of the 
influenee of these factors on the use of data by administrators at the college and their own use of 
data. 

4,3 Discussion of Correlates 

These findings are encouraging for initiatives designed to bring about changes in behavior 
among members of an institution. Again, while the findings do not establish a eausal 
relationship, we did see a few persistent correlations that are compelling. For example, faculty 
members and administrators participating in Achieving the Dream were much more likely than 
non-participating faculty to make more frequent use of data that are of particular interest to 
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Achieving the Dream. The survey also asked faculty and administrators about various possible 
reasons they might not use data on students. We found that the reasons most associated with a 
lower tendency to use data were that using student data was not a job responsibility or that the 
data were irrelevant to their jobs. Colleges engaged in such an initiative may thus want to discuss 
the role that reviewing student data plays in faculty and administrators’ responsibilities and may 
want to clarify the ways in which the available data are relevant to various jobs at the college. 

Also notable were the differences in frequency of data use on student outcomes, participation in 
organized discussions on student achievement, use of data and research in teaching-related 
discussions, and use of disaggregated data by faculty and administrators of different 
races/ethnicities. 

The differences in data use by department, in particular, are interesting. We noted that faculty 
who teach in developmental or for-credit occupational programs were more likely to use data and 
research to inform teaching related practices than those in other fields. And we noted that faculty 
in non-credit occupational programs used data related to student outcomes more frequently than 
did faculty in other fields. The finding for developmental faculty is consistent with the focus of 
Achieving the Dream. It may be that the findings for the for-credit occupation programs result 
from longstanding licensure and certification requirements — data on these credentials may 
impact what these faculty do in the classroom. Our findings related to non-credit occupational 
programs may be the result of these faculty acting in a more entrepreneurial and market-oriented 
fashion as they create course offerings. In doing so, they may rely more heavily on data. 

The fact that the correlation between academic department practices and faculty practices was 
stronger than that between the college’s overall use of data and the faculty’s use of data was 
unexpected and requires more exploration. In addition, we hypothesized that we would see a 
meaningful positive relationship between administrators’ use of data and administrators who 
responded that their college engaged in program evaluation and strategic planning, required the 
departments and divisions to set measurable goals as part of the planning process, and tied 
budget requests to evidence that students will benefit. Instead, we found that these practices are 
associated with a lower frequency of use among administrators. Additional analyses are needed 
to better understand these relationships. 

Finally, as we discussed in our methodology section, we selected the states based on a 
conception that there was significant pressure on the colleges from accrediting agencies and state 
policy to use data to improve student academic success. We hypothesized that this pressure 
would drive administrators to use data. However, we found only a very weak correlation (< 0.1) 
between administrators’ ratings of the influence of these factors on the use of data by 
administrators at the college and their own use of data. 
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5. Patterns of Data Use by College 



In this section, we report on our examination of the amount of data use among the Achieving the 
Dream colleges in our sample. To do this, we ranked the colleges according to their average 
scores on the four indicators of data use by faculty and three indicators of data use by 
administrators. Recall that the indicators of data use that we developed are: 1) use of data on 
student outcomes; 2) use of data disaggregated by student race, ethnicity, or income; 3) 
participation in organized discussions on improving student success; and 4) use (by faculty) of 
data and research in teaching-related decisions. We did a similar ranking by the round in which 
colleges joined Achieving the Dream (Round 1 or Round 3), and the state in which they are 
located. Our aim was to determine if there is consistency in the ranking of colleges by the 
various indicators of data use. If we did find some degree of consistency in the ranking of 
colleges by these indicators, this would give us confidence to use them as measures of the 
amount of data use at the institution level. It would allow us to examine the relationship between 
college-level indicators of the amount of data use and measures of institutional performance. We 
could also examine the practices of colleges with higher levels of data use by faculty and 
administrators with those where the amount of data use is lower. To the extent that there is not 
consistency in the ranking of colleges by these various measures, it would be difficult to use 
these indicators appropriately for comparison. 

In Appendix D, Table D.l.a shows the mean values (after conversion to z-scores) for each 
college in the sample of the four indicators of data use by faculty and the three indicators of data 
use by administrators. Note the considerable variation in the mean responses across the colleges. 

To facilitate comparison. Table D.l .b presents rankings of the colleges by the mean value of 
each indicator. Note that College 25 in North Carolina ranks first among the colleges on the first 
three indicators and fourth on the indicator of the extent to which faculty at the college use data 
and research in teaching-related decisions. A couple other colleges show some consistency in 
their ranks across the four indicators. For example. College 19 ranks sixth in the average 
frequency with which faculty use data on student outcomes; third in the extent to which faculty 
use student data disaggregated by race, ethnicity, or income; ninth in the frequency of 
participation by faculty in organized discussions on improving student outcomes; and eighth in 
the extent to which faculty use data and research in decisions about teaching. At the same time, 
however, there are many cases where a college’s ranking varies considerably across the different 
indicators. For example. College 34 ranks third on faculty indicator 1, thirty-fifth on faculty 
indicator 2, thirty-first on indicator 3, and eighteenth on indicator 4. 

A similar lack of consistency in rankings is evident across the indicators of data use by 
administrators. Moreover, the rankings of colleges by the comparable indicators for faculty and 
administrators also do not in general follow a clear pattern. For example. College 26 ranks 
second in the indicator of the frequency with which faculty use data on student outcomes and 
thirty-third on the comparable measures for administrators. This is consistent with the finding 
from the previous section that there does not seem to be a strong correlation between the extent 
to which faculty use data and research and their perceptions of the extent to which the college 
generally uses data for decision making. 
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Another finding from the previous section is that faculty and administrators who participated in 
Achieving the Dream are more likely than those who had not participated to use data according 
to the four indicators. Given this, we hypothesized that colleges where a greater percentage of 
faculty or administrators were involved in Achieving the Dream would on average exhibit higher 
rates of data use. From Table D.l.b, we see that College 25 had the highest percentage of faculty 
respondents who were involved in Achieving the Dream, and it ranks first on three of the 
indicators of faculty data use. However, College 40 is ranked second in the percentage of faculty 
respondents who participated in Achieving the Dream, but it ranks thirty-fifth, tenth, eighteenth, 
and thirtieth, respectively, on the four indicators of faculty data use. A similar lack of ra nk order 
consistency is evident between the measures of the extent to which administrators were involved 
in Achieving the Dream and indicators of data use by administrators. 

We hypothesized that it would be easier to engage faculty and administrators in using data for 
decision making at smaller colleges compared to larger ones because of the challenges of 
communication and coordination in large organizations. However, as is evident from Table 
D.l.b, there does not seem to be a clear correlation between institutional size (measured by FTE 
enrollment) and indicators of data use by faculty and administrators. 

Table D.2.a shows the mean institutional-level values for the faculty and administrator indicators 
by state and by the round in which colleges joined Achieving the Dream. Table D.2.b ranks the 
states and rounds by the mean values in Table D.2.a. 

From these tables we see that colleges in North Carolina rank higher than those from other states 
in the average values for the first three indicators of faculty data use, but third on the indicator of 
the extent to which faculty use data on students in teaching-related decisions. The Achieving the 
Dream colleges in Texas rank highest (on average) on all three indicators of administrator data 
use. However, there is a fair amount of inconsistency in the ranking of the other states across the 
various indicators of data use for both faculty and administrators. 

The colleges that joined Achieving the Dream in the first round on average exhibit higher rates 
than those in the third round of data use on all of the faculty and administrator indicators, with 
the exception of the measure of the extent to which faculty participate in organized discussions 
on improving student success. This is consistent with the hypothesis that colleges that have been 
involved in Achieving the Dream longer will be more advanced in their use of data for 
improving student success. However, these findings are merely suggestive; they cannot be seen 
as definitive evidence of a causal relationship between Achieving the Dream and more extensive 
use of data for improvement. 
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6. Findings from Telephone Interviews 



This section presents findings from telephone interviews of faculty and administrators at four of 
the Achieving the Dream community colleges that participated in the survey. In total, we 
interviewed 17 faculty members and 25 administrators/staff. The interviewees included a mix of 
individuals who had and had not been directly involved with the Achieving the Dream initiative. 
The faculty we interviewed taught either developmental or college-level math or English. The 
senior administrators we interviewed included, at each college, the senior academic officer, the 
senior student services officer, the director of institutional research, the Achieving the Dream 
coordinator, and at least one member of the student services staff. 

We analyzed the interview transcripts to look for themes that could further explain the survey 
findings. In particular, in this section we address data use at the colleges and the connection (or 
absence of a connection) between strategic planning and budgeting and the use of data by faculty 
and administrators. 

6.1 Use of Data by Faculty and Administrators 

In general, the telephone interviews we conducted reinforce the survey findings about the use of 
data. Faculty and administrators reported that they reviewed and used student outcomes data on a 
regular basis. In the survey we asked specifically about grades, placement scores, retention rates, 
transfer rates, and graduation rates. We also asked about research carried out by the college and 
about college-based or outside research on effective practices. In more open-ended discussions 
about data use in the interviews, we heard many faculty refer, not surprisingly, to grades and 
informal in-class assessments of student comprehension of course material as measures of 
student performance. To a more limited extent, faculty also spoke about using placement scores 
and scores on other tests to gauge the preparedness of their students and to understand the range 
of skills they possessed. The administrators we interviewed generally discussed the use of data 
on enrollments, retention rates, and graduation rates. Administrators also often mentioned data 
types that we did not ask about in the survey, including information that supports course and 
faculty scheduling, tracking data on tutoring center usage, and health services statistics. 

Faculty interviewees at three of the four colleges reported that they regularly met with colleagues 
to discuss student achievement in their courses and to compare perceptions and trends about 
student progression. Two interviewees reported on how such discussions eventually facilitated a 
change in a specific practice (in one case, the result was a faculty-wide push to improve critical 
thinking skills among students across the curriculum; in the other case, the result was the 
prohibition of the use of calculators in developmental math courses). While the evidence used 
during the discussions that led to these changes reportedly emphasized first-hand observations of 
students more than numerical data, in both cases placement test scores were also used to 
understand the level of students who were entering classes. 

We heard a range of perspectives on the role of faculty in using student outcomes data. 

According to the survey results, only 13% of faculty did not use data and research on students 
because they believed it was not part of their responsibilities. While many faculty interviewees 
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indeed argued that faculty should be actively involved in data collection and analysis efforts, 
several remarks made by others suggest that among other faculty there is a strong disinclination 
to collect or use such data. In one interview, for example, a professor who did not think that the 
use of data and research was part of a faculty member’s responsibilities questioned the value of 
data, arguing that many faculty members, who as teachers have nearly daily exposure to their 
students in classroom settings, are in a good position to understand those students’ needs without 
recourse to additional data. 

The interview findings suggest that administrators and faculty who sit on committees may be 
more likely than others to use large-scale surveys and tests. For example, administrator 
interviewees at two colleges cited the use of internally developed surveys to gather feedback 
from faculty and students on their opinion of areas needing improvement. Interviewees from 
three colleges noted the use of external instruments, including CCSSE, Noel-Levitz, or the 
University of Texas’ Learning and Study Strategy Inventory (LASSI), to identify areas of 
improvement. Using LASSI data as well as registration and course completion data and grades, 
one college developed a model of students at risk showing that students who fail or drop out of 
the same course twice have very low success rates. They then developed a plan requiring that 
such students see an academic advisor for the provision of additional supports and services prior 
to registration for additional courses. 

It is clear that all four of the colleges involved in the telephone interviews make use of a variety 
of data sources, but the extent to which they are using data they themselves collect to improve 
practices varies. We did hear from interviewees at one college about their having implemented 
evaluations of strategies they pursued and about how they have now acquired preliminary 
evidence suggesting that their strategies result in improved learning outcomes. Interviewees at 
another college discussed the goal of using data to inform classroom instruction. They have 
developed software that allows faculty (who choose to do so) to enter course objectives and 
whether they meet those objectives. The ultimate impact of having that information and whether 
it results in changes in classroom practices remains to be seen. We heard little from our 
interviews with representatives from any of the colleges suggesting that analysis of internally 
collected data is college-wide. 

Three of the four colleges are using “dashboards.” These dashboards provide current statistics to 
senior administrators on academic indicators such as enrollment and retention as well as other 
kinds of information, including student visits to the college’s health services unit for specific 
reasons. The use of dashboards and fact books as sources of data contributes to a perception held 
by many of the interviewees that their college has established a culture of evidence. Lor example, 
with respect to their fact book, one administrator commented that “it is just a wealth of 
information for us. Now we are looking at that and saying, ‘Okay, I see that our graduation rates 
have started to slip a little bit.’ And we start talking about the data and why and what we can do 
to maybe change the course of that. So, yes, I would say that ‘data-driven’ is permeated 
throughout the college.” These dashboards represent a significant change in information access. 
Lirst, they permit access to a wider range of information to a wider range of people at the 
college. Second, they provide current, sometimes in real time and sometimes daily, updates of 



^ LASSI is a tool developed at the University of Texas at Austin that assesses students’ “awareness about and use of 
learning and study strategies related to skill, will and self-regulation eomponents of strategie learning.” 
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key college indicators. Third, they represent a form of data that is “pushed out” to people at the 
college, rather than having individuals retrieve the information themselves. Clearly, as the 
quotations above indicate, the dashboards provide information on which administrators can act. 
Yet it is still unclear whether these dashboards are part of a continuous improvement process that 
involves resolving issues and evaluating solutions or whether they just provide information on 
college’s performance that is not systematically used in decision making about the management 
of programs and services. 

In general, the absence of more extensive and more complex data analysis may in part be due to 
problems of time, resources, and expertise. Although the survey results indicate that only 32% of 
faculty and only 9% of administrators are too busy to use data and research on students, a 
number of interviewees made comments about either not having the time to use the considerable 
amount of data available at the college or not having the expertise. For example, one interviewee 
expressed interest in comparing the performance of students who have and have not received 
tutoring in courses that they take after participating in that intervention. However, she has not 
had the time to pursue this interest. 

Interviewees also said that they rely on their institutional research office to help them interpret 
the data they do receive, although at least one senior administrator at each college reported 
extracting and analyzing data themselves. A reliance on the IR staff can be problematic because 
they are generally very busy and may not have the time to help everyone analyze relevant data. 
Many of the faculty and administrators we spoke with said they did not feel comfortable 
analyzing data or even know what questions to ask. This is inconsistent with our survey findings 
in which only 17% of faculty and 1 1% of administrators indicated not having the skills to 
analyze data. 

6.2 The Relationship Between Strategic Planning and Budgeting and Use of Data 

One surprising finding from the survey was the weak correlation between the extent to which 
administrators and faculty used data on student outcomes and whether or not they indicated that 
their college overall uses data to evaluate programs or to make program-related decisions. 
Similarly, the frequency with which administrators used data did not seem to be related to 
whether or not college departments were required to set measurable goals as part of the planning 
process or whether or not the college required that budget requests be supported by evidence that 
students would benefit. We expected that faculty and administrators in colleges where program 
review, strategic planning, and budgeting are informed by data on students would be more likely 
to use data themselves. Our telephone interviews suggest a possible explanation why we did not 
find this to be the case. 

The respondents to our telephone interviews indicated that involvement in strategic planning and 
budgeting is at their colleges generally limited. When faculty and lower-level staff are involved, 
it seems as though they are still far removed from the final budgeting decisions and setting of 
priorities. For example, while some colleges use large strategic planning teams that do involve 
personnel from across the college, it is not clear what influence these teams have on final budget 
allocations. Another model can be likened to a pyramid in which plans get rolled up to the 
leadership group, which ultimately makes budget decisions. One interviewee described the 
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process as follows: “Every two years we develop a set of improvement objectives for every unit, 
and it sort of pyramids up so that all the units come together with the organizational entity above 
them and they come up with another prioritized list. And so, ultimately you end up with 
prioritized lists that are used to get funding. So if you want to get funded in the budget process, 
you have to show that it’s in your plan, which means you have to show that there’s some data to 
support the request, . . . that’s how it gets linked. So you can’t get funded without a plan.” This 
college has clearly sought to use data to inform planning and budgeting, but it is not clear how 
the data inform the final decisions. It may be true that departments and divisions cannot get 
funded without a plan (and data), but it is not clear what role data play in how funding decisions 
are made vis-a-vis a number of competing plans. 

It is still a challenge for colleges to fully integrate their planning and budgeting processes and 
use data to support those processes. Although there may be much talk about the use of data to 
inform budgeting, our interviews suggest that the reality is usually quite different. One 
administrator interviewee commented that although his college’s administration is trying to tie 
the budget and strategic planning processes together, “it’s a chore to try to get planning ahead of 
budgeting.” He added that budgeting has “not aligned itself very well with strategic planning.” 

These examples may help explain why the survey findings indicate that there is little correlation 
between data use by individual faculty and administrators and strategic planning and budgeting 
by the college. Interviewees at all the colleges stated that they have integrated planning and 
budgeting so that the two processes do not occur separately and that the college’s priorities direct 
the budget, but they also qualified those statements, explaining that they were still improving the 
process and were working on further aligning planning and budgeting. 

One area where there was clear consensus on the use of data was in making decisions about 
hiring practices, mostly as it relates to the ability to fund a new position or to override a 
restriction in terms of a hiring freeze. Data, such as enrollment trends, are needed to justify a new 
position or why a vacant position must be filled. 

Findings from the interviews suggest that data use for program review is mixed. It appears that 
there may be a lot of unfocused data use in program reviews. For example, one faculty member 
commented, “When the committees or departments present the program review, oh my 
goodness, [there] is so much data in there.... [Tjhere is too much information. Information 
overload.... It’s just all these charts....” Another professor said that he didn’t see overall 
program assessment as his job; he felt that the student outcomes for his courses were where his 
responsibilities ended. 
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7. Conclusion 



This report presents findings from what is to our knowledge the most extensive survey to date of 
the use of data and research for decision-making by faculty and administrators at community 
colleges. It is based on surveys of faculty and administrators at 41 colleges involved with 
Achieving the Dream as well as telephone interviews with faculty and administrators at four of 
those same colleges. Achieving the Dream is a national initiative designed to improve outcomes 
for community college students, particularly those who have traditionally faced barriers to 
success in college. One of the key principles of Achieving the Dream is that for colleges to 
improve student outcomes on a substantial scale, they need to engage faculty, administrators, and 
staff on a wide scale in examining data on how students perform and in using that information to 
devise strategies for addressing gaps in attainment among student groups. Colleges should seek 
to build a “culture of evidence” in which decisions about how to organize, manage, and fund 
instruction and student support services are made based on evidence of what works to promote 
student success. 

The survey showed that relatively high proportions of the faculty and administrators in the 
Achieving the Dream colleges surveyed used data on student outcomes, and the telephone 
interviews support this finding. More than half of faculty members reviewed or used data on 
placement test scores, retention rates, graduations rates, and measures of student learning other 
than grades at least once a year. About half of faculty and administrators used data on student 
achievement gaps on an annual basis. About two thirds of faculty and administrators used 
outside research on effective practices at least annually. 

The majority of faculty at the Achieving the Dream colleges surveyed used data and research at 
least to some extent in decisions related to teaching. And most faculty and administrators 
surveyed indicated that they participate at least once a year in organized discussions on 
improving student success. Two thirds of faculty agreed that asking faculty to regularly 
participate in discussions about data on student outcomes is a good use of their time. Only 13% 
of faculty members indicated that using data and research on students is not part of their 
responsibilities as a faculty member, though the telephone interviews suggest that resistance to 
using data may be strong among some faculty. 

The high rate of data use at the colleges surveyed is perhaps not surprising, given their 
involvement in an initiative premised on the importance of broadly engaging faculty and staff in 
using data and research to improve the quality of programs and services. Indeed, we found that 
faculty and administrators who were involved in Achieving the Dream used data on student 
outcomes more frequently and participated in organized discussions on improving student 
outcomes much more often than did their colleagues who were not involved in the initiative. 

At the same time, not all participants in Achieving the Dream were heavy data users, and more 
generally, there was considerable variation among individual faculty and administrators in the 
extent to which they used student data. For example, over a third of faculty never used measures 
of student learning other than grades or looked at information on students broken down by race 
or ethnicity, and nearly one third never reviewed data on student achievement gaps. And the 
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telephone interviews suggest that, despite survey findings that seem to indicate otherwise, many 
faculty and administrators may feel that they lack the skills to analyze data in ways meaningful 
to their jobs. 

Female faculty members indicated that they used data and research in teaching-related decisions 
more frequently than did male faculty members, and they participated in organized discussions 
about students more frequently than did their male counterparts. White faculty were less likely 
than non- White faculty to use data on student outcomes generally, to use data in teaching-related 
decisions more specifically, and to participate in organized discussions on student achievement. 
Faculty who classified themselves as Black and Native American used data disaggregated by 
race, ethnicity, and income more frequently than did faculty of other races and ethnicities. 
Interestingly, in contrast to patterns observed for faculty. White administrators were more 
frequent users of student outcomes data than were non- White administrators. Contrary to 
expectation, we did not find much of a correlation between a faculty member’s rank or the length 
of time a faculty member had been at the college and the extent to which he or she used data. 

We found that the practices of faculty members’ departments were more strongly associated with 
their tendency to use or not use data than were practices and values of the college overall. Not 
surprisingly, faculty in departments that used data in departmental decision-making and that met 
frequently as a department to discuss student success rates and evaluate the effectiveness of 
programs were more likely to use data as measured by the four indicators of data use that we 
employed. In contrast, there is only a weak correlation between the various indicators of data use 
and the extent to which respondents indicated that their college overall uses data on outcomes to 
evaluate programs. 

There is also surprisingly little correlation between the extent to which administrators said that 
their college uses data for program-related decisions and the frequency with which they 
themselves used data. Similarly, there does not seem to be much correlation between the 
frequency with which administrators used student outcomes data or engaged in organized 
discussions on improving student outcomes and whether or not their college requires 
departments and divisions to set measurable goals and objectives as part of the planning process. 
The telephone interviews suggest that colleges continue to be challenged in their efforts to fully 
integrate their planning and budgeting processes and encourage widespread use of data to 
support those processes. 

The apparent disconnect between the extent of data use by faculty and administrators and the 
views and management practices of the college’s leadership calls into question a central premise 
of Achieving the Dream — that commitment by a college’s leadership and the way that a college 
approaches program evaluation, strategic planning, and budgeting are key to encouraging the use 
of data for improvement by college personnel. Our findings suggest that leadership commitment 
and a data-oriented approach to institutional management may not be sufficient to encourage 
faculty and administrators to become more data-oriented in practice. Additional efforts at the 
department level are probably needed to change the behavior of faculty in particular. Indeed, we 
found that faculty in developmental education departments and for-credit occupational programs 
were more frequent users of data than were faculty in other types of departments, particularly 
those in general education. The greater intensity of data use in developmental education 
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departments is perhaps not surprising given that improving developmental instruction has been a 
major focus of Achieving the Dream. The baseline evaluation of the first-round Achieving the 
Dream colleges found that the vast majority of participating colleges, if not all them, were 
implementing some sort of strategy aimed at improving developmental outcomes (Brock et ah, 
2007). It may well be that a similar intensive focus on improving outcomes is needed to change 
practices and to influence the culture in other types of departments. 

In examining the relative extent of data use among the colleges in our sample, we did not find 
much consistency in the rankings of these colleges by institution-level averages of the various 
indicators of data use by faculty and administrators. This suggests that colleges in the sample 
differed in the types of data they emphasized. Contrary to expectation, faculty and administrators 
at larger colleges were not, on average, heavier users of data than faculty and administrators at 
smaller colleges. 

Although colleges with higher levels of participation in Achieving the Dream by faculty and 
administrators did not exhibit higher average rates of data use, we did find that colleges that 
joined Achieving the Dream earlier (in Round 1 rather than Round 3) had higher rates of data use 
on all but one of the faculty and administrator measures. This is consistent with (although it does 
not prove) the hypothesis that colleges that have been involved in Achieving the Dream longer 
are more advanced in their use of data for improving student success. It also suggests that 
engaging faculty and staff in using data and building a culture of evidence is a complicated 
process that requires concerted effort over a long period of time. 

The findings from this study suggest three broad conclusions and one point for further analysis. 
First, the survey findings suggest that Achieving the Dream may have had an impact on data use 
at the colleges. The data use may not be as widespread as had been hoped or as integrated with 
planning and budgeting functions as might be expected, but the greater use of data on student 
outcomes by faculty and administrators who are involved in the initiative suggests that an 
externally originated initiative can bring about changes in practice. This is promising for 
Achieving the Dream and other initiatives seeking organizational change to improve institutional 
performance and student outcomes. As mentioned, CCRC and MDRC plan to survey a panel of 
faculty and administrators who participated in this one again in 201 1. This will provide more 
definitive evidence about whether or not Achieving the Dream is helping to foster increased use 
of data in decision making. 

Second, the findings suggest that producing substantive changes in culture and practice is a long 
process. The Achieving the Dream initiative began as a five-year effort. Our finding that the 
Round 1 colleges, which joined in 2005, tended to have higher rates of data use than the Round 3 
colleges, which joined in 2007, indicates that several years of effort may be needed before 
changes in behavior are institutionalized. The follow-up survey planned for 2011 will provide a 
clearer indication of how long this change process takes. 

Third, the apparent disconnect between the extent of data use by faculty and administrators and 
the views and management practices of the college’s leadership suggests that leadership 
commitment and a data-oriented approach to institutional management may not be sufficient to 
encourage faculty and administrators to become more data-oriented in practice. Greater emphasis 
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may be needed at the department level to encourage the use of data for improvement. This 
implies that the notion of broad engagement in the process of analyzing student data is 
particularly important, especially for department and division chairs at the college. 

Finally, further analysis is needed to better understand the relationship between data use and 
budgeting and planning efforts. Our finding that colleges with established systems for strategic 
planning and budgeting were less likely to use data disaggregated by race or income than 
colleges without such systems is counterintuitive; additional analysis may shed more light on 
this. 
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Appendix A: Methodology for Creating Indicators of 
Data Use and Correlative Factor Measures 



Indicators of Data Use 



To create the four indicators of data use, we converted the responses to survey items related to 
each indicator to z-scores and then averaged the scores for each indicator and respondent.^ So, 
for example, the value for each survey respondent of the first indicator on the use of student 
outcomes data was calculated by averaging the scalar variable responses to sub-questions on the 
frequency of use of data by the respondent on each of the following outcomes: measures of 
student learning, retention rates, graduation rates, transfer rates, developmental education 
completion rates, outside surveys such as CCSSE or Noel-Levitz, and student achievement gaps. 

The four indicators, along with the items used to create each, are as follows: 

1. Use of data on student outcomes 

• Frequency of use of data on student outcomes, including: 

o Measures of student learning 
o Retention rates 
o Graduation rates 
o Transfer rates 

o Developmental completion rates 

o Outside surveys such as the Community College Survey of Student Engagement 
(CCSSE) andNoel-Eevitz 
o Measures of student achievement gaps. 

2. Use of data disaggregated by student race/ethnicity or income 

• Frequency of use of data disaggregated by: 

o Students’ race or ethnicity 
o Students’ income or financial aid status. 

2. Participation in organized discussions on improving student outcomes 

• Frequency of participation in organized discussions on: 

o Improving academic achievement or closing achievement gaps 
o Academic needs or performance of students of color 
o Academic needs or performance of low-income students. 



^ We also created the indicators using factor analysis. Because the factor scores created through this analysis 
produced results very similar to the average z-score composite indicators, we decided to use the latter, given their 
relative simplicity and transparency. 
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4. Use (by faculty) of data and research in teaching-related decisions. 

• Use (by faculty) of data and research in decisions related to: 

o Curriculum 
o Teaching methods 
o Advising students 

o Identifying students who are struggling academically. 

Correlative Factor Measures 



The following are definitions by category of the measures developed for factors we hypothesized 
to be correlated with use of data and participation in discussions about improving student success 
by faculty and administrators. Some of these factors are composite measures created by 
averaging z-scores for each sub-item indicated below each in bullets (in the same way that we 
created the indicators of data use). 

Involvement in Achieving the Dream 

• Respondent involved in the Achieving the Dream core or data team (Y/N). 

• Respondent involved in implementing Achieving the Dream strategies (Y/N). 

• Respondent involved in either way (Y/N). 

Perceived usefulness of data related to student outcomes 

• Perceived usefulness of data related to student outcomes: 

o Measures of student learning 
o Retention rates 
o Graduation rates 
o Transfer rates 

o Developmental completion rates 

o Outside surveys such as the Community College Survey of Student Engagement 
(CCSSE) andNoel-Eevitz 
o Measures of student achievement gaps. 

• Eevel of agreement about usefulness of IR to teaching (faculty only). 

Perceived accessibility and quality of data and research on students 

• Perceived accessibility and quality of college data and research: 

o Data are accurate 
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o Data format is user friendly 
o IR is response to requests for information 
o IR is sufficiently staffed 
o IR reports are clear 
o Reports from IR are timely. 

Reasons for not using student data 

• Extent to which the following are reasons respondent does not use data: 

o Respondent is too busy 
o Not respondent’s responsibility 
o Doesn’t have the skills 
o Doesn’t trust the data 
o Data are irrelevant to respondent’s job. 

• Level of agreement about whether asking faculty to participate in organized discussions is 
good use of their time (faculty only). 

Attitudes about the potential of students to succeed and the college ’s responsibility in helping 

students succeed 

• Extent to which respondent believes that with adequate support, most students can succeed 
academically (as opposed to believing that some students are bound to fail, no matter what 
the college does). 

• Extent to which respondent believes that it is often appropriate for the college to give extra 
services or help to students on the basis of race, ethnicity, or income (as opposed to 
believing that it is never appropriate for the college to do so). 

• Extent to which respondent believes that s/he and her/his colleagues need to try different 
approaches to help low-achieving students succeed (versus believing that they are doing all 
they can to help low-achieving students). 

Participation in training on data analysis, assessment, or program evaluation 

• Respondent has participated in training or professional development on IR or data analysis 
(Y/N). 

• Respondent has participated in training or professional development on program evaluation 
or assessment (Y/N). 

Academic department practices (faculty survey only) 

• Extent to which department uses data and research in decisions regarding: 
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o Curriculum 
o Teaching practices 
o Tutoring or other aeademic support 
o Program planning 

o Academic program review or evaluation 
o Long-term strategic planning 
o Budget and resouree alloeation 
o Identifying and redesigning high-failure rate eourses 
o Identifying areas for improvement. 

• Faeulty in department meets at least onee a year to discuss student suecess (Y/N). 

• Department regularly evaluates effectiveness of programs (Y/N). 

College/administrative department practices 

• Perceived extent to whieh college uses data on student outcomes to evaluate programs and 
praetices (faculty survey only). 

• Pereeived extent of eollege leadership’s commitment to using data to make deeisions. 

• Perceived clarity of leadership’s vision on how to increase student academic success. 

• Perceived extent to which college uses data and research in decisions regarding 
(administrator survey only): 

o Curriculum 
o Program planning 

o Academic program review or evaluation 
o Long-term strategic planning 
o Budget and resource alloeation 
o Identifying areas for improvement. 

• College evaluates the effeetiveness of educational programs and services (Y/N) 
(administrator survey only). 

• College uses data on student outcomes to evaluate aeademie programs and services (Y/N) 
(administrator survey only). 

• Administrators regularly use data on student outeomes broken down by raee/ethnieity and 
income (Y/N) (administrator survey only). 

• Perceived extent to whieh administrators, faculty, and staff at college work eollaboratively 
to improve programs and services. 
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College has a strategie plan that is used to guide operational planning (Y/N) (administrator 
survey only). 



• Departments and divisions required to set measurable goals and objeetives as part of the 
planning process (Y/N) (administrator survey only). 

• College requires that budget requests be supported by evidence that students will benefit as 
a result (Y/N) (administrator survey only). 

Influence of external factors in promoting data use by college administrators (administrator 
survey only) 

• Perceived extent of influence in promoting use of data by college administrators of the 
following factors: 

o College leadership 
o Board of trustees 
o State mandates 
o Accreditation reviews 
o Achieving the Dream 
o Other grant programs. 

Academic program area (for faculty) or administrative functional area (for administrators) 

• Faculty program area: 

o Developmental 
o General education 
o For-credit occupational 
o Not-for-credit occupational 
o Adult basic education 
o Continuing education 
o Other. 

• Faculty member also has an administrative role (Y/N). 

• Administrator functional area: 

o Academic affairs 
o Admissions 
o Athletics 

o Business and industry 
o Career and technical education 
o Continuing education 
o External relations 
o Finance 
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o Human resources 

o Institutional research/effectiveness and planning 
o President’s office 
o Registration 
o Student affairs/services 
o Other. 

• Administrator also a faculty member? 

Length of time at the college and rank (for faculty) 

• Length of time at the college: 

o Less than 1 year 
o 1 to 5 years 
o 6 to 10 years 
o More than 10 years. 

o 

• Rank (for faculty only): 

o Full professor 
o Associate professor 
o Assistant professor 
o Instructor (full-time) 
o Instructor (part-time or adjunct). 

Respondent demographics 

• Age. 

• Gender. 

• Race/ethnicity. 
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Appendix B: Response Rate by College 



College 


Faculty Response and 
AtD Involvement 


Administrator Response and AtD 
Involvement 


Response 
rate (%) 


Involved 
in AtD 
(%) 


Not involved 
in AtD (%) 


Response 
rate (%) 


Involved 
in AtD 
(%) 


Not involved 
in AtD (%) 


College 1 


73% 


33% 


67% 


75% 


67% 


33% 


College 2 


63% 


40% 


60% 


80% 


50% 


50% 


College 3 


64% 


47% 


53% 


71% 


100% 


0% 


College 4 


54% 


39% 


61% 


70% 


57% 


43% 


College 5 


67% 


54% 


46% 


75% 


67% 


33% 


College 7 


74% 


15% 


85% 


93% 


52% 


48% 


College 8 


58% 


46% 


54% 


79% 


64% 


36% 
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Appendix C: Demographics of Respondents 
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Table C.4: Respondents length of time at the college 

Faculty Administrators 

One to More 

Less than 1 One to Six to 10 More than 10 Less than 1 five Six to 10 than 10 Total 

year five years years years Total (N) year years years years (N) 
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Table D.l.a: Indicators of data use: Mean z-scores by college 

Involvement in 
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Table D.l.b: Indicators of data use: Ranking of Achieving the Dream colleges by mean z-scores 

Involvement in 

AtD Data Use Indicators: Faculty Data Use Indicators: Administrators 
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Table D.2.b: Indicators of data use: Ranking of Achieving the Dream states and rounds by mean z-scores 



S 

■o 



^ s 15 s 

o . N .2 

« tC S 2 

s 3 W) o 

c ^ © -o 



© 

o 



-o 

© 

&x 

© ^ 



&x . 

3 



*o *0 
C 

NH © 



^ ct 

^ *0 

® ® <u 

« ^ s 

o ^ is 

^ © 



© 



&x 



© ct 
© 



.3 © 



ct 

*-3 .= 



2 c -o 



© 

o 



fi C 

■-. I 

Vi 



ct S 
!5X O 

© *rt 



© 

© *© 
© 



fN 

© 

^© 

-S 



ct 

GX ^ 

Ct 

Vi 



, © 

© © c 
« " 2 « 






© © 

3 5 

'M © 



^ ro 1^ 



(N ^ ^ 



(N ro 1^ 



(N ro ^ 



m 



1^ 



m 



1^ 



(N 



J u S 
^ ^ 






lO (N 



<N 



1^ ro 



<N 



lO ^ 



<N 






<N 






lO 






1^ (N 



<N 



lO 






^ (N 



(N 



^ ho 



ho ro 



<N 



> 









t3 

C3 

o 

Pi 



t3 

C3 

o 

Pi 



Achieving the Dream Survey Report 







